# NOSIBLE World S3 Guide

This guide is for users consuming the private NOSIBLE World S3 distribution. It
explains where the files live, what each artifact is, and how to read the four
event slice artifacts.

Scope: this is the full NOSIBLE World event distribution for the covered dates.
Aggregate statistics (event counts, coverage, and the country, language, or
sentiment mix) reflect the full event universe for those dates.

The bucket is:

```text
s3://nosible-world/
```

It holds two versions of the NOSIBLE World database, each under its own prefix
and partitioned by slice date:

```text
s3://nosible-world/slices/<YYYY-MM-DD>/         # NOSIBLE World v1.1 (previous)
s3://nosible-world/slices_v1_2/<YYYY-MM-DD>/    # NOSIBLE World v1.2 (current)
```

You have read access to both prefixes. v1.2 is the current version and is
recommended for new work; v1.1 is the previous version, retained for continuity.
Both versions use the identical four-artifact layout and the same lookup
contracts, reader code, and column and field definitions described in this
guide. Everything below that references `slices_v1_2/` applies equally to
`slices/` -- just substitute the prefix.

The bucket root also holds this guide and a v1.2 sample file:

```text
s3://nosible-world/NOSIBLE_WORLD.md     # this guide
s3://nosible-world/samples_v1_2.json    # sample events from v1.2
```

Each date prefix contains exactly four objects:

```text
events.ndjson
events.locs
events_oai.hnsw
events_map.ipc
```

The date range is:

```text
2015-01-01 through 2026-07-01, inclusive
```

The bucket is private. Users receive individual IAM access keys with read-only
access to the `slices/` and `slices_v1_2/` prefixes (the four object names below
in each date directory) and to the `NOSIBLE_WORLD.md` and `samples_v1_2.json`
files at the bucket root.

## Documentation and Samples

- `NOSIBLE_WORLD.md` -- this guide, at the bucket root: where the data lives,
  what each artifact is, the lookup contracts, and reader code.
- `samples_v1_2.json` -- a small, human-readable sample set at the bucket root
  (`s3://nosible-world/samples_v1_2.json`). It is a pretty-printed JSON array of
  complete event records, and every element is exactly one `events.ndjson` line
  (the full `event`, `signals`, `coverage`, `entities`, `tickers`, `ontology`,
  `provenance`, and `oai_vector` payload). Use it to inspect the event shape
  without pulling a slice; the authoritative data is the S3 slice distribution
  described below.

## The Four Artifacts

`events.ndjson` is the canonical event store. It contains one complete JSON
event object per line. Use it for full event payloads, provenance, entities,
tickers, ontology blocks, coverage, similar events, signals, coordinates, and
the per-event OpenAI embedding string.

`events.locs` is the random-access sidecar for `events.ndjson`. It is a NumPy
`.npy` file containing a one-dimensional little-endian `int64` array of
cumulative byte offsets. If `locs[i] == 1234` and `locs[i + 1] == 1600`, event
line `i` lives at byte slice `[1234:1600]` in `events.ndjson`. The final entry
is a sentinel equal to the total byte size of `events.ndjson`.

`events_map.ipc` is the flat filter, sort, and facet table. It is an Arrow IPC
file with one row per event. Use it for fast scans over columns like country,
language, sentiment, materiality, coverage, geography, IAB, IPTC, GICS,
NOSIBLE taxonomy labels, `has_tickers`, and `event_id_u64`.

`events_oai.hnsw` is the semantic-search index. It is a uSearch HNSW index over
OpenAI `text-embedding-3-large` event embeddings: `3072` dimensions, cosine
metric, `f32`. Search hits return integer keys. Those keys are
`xxhash64(event_id, seed=0)` and match the `event_id_u64` column in
`events_map.ipc`.

## Access

Configure the AWS CLI with the access key supplied for your organization:

```bash
aws configure --profile nosible-world
```

Use region:

```text
us-east-1
```

Verify the credential:

```bash
aws sts get-caller-identity --profile nosible-world
```

List the two version prefixes:

```bash
aws s3 ls s3://nosible-world/ --profile nosible-world
```

List available date prefixes (v1.2 shown; use `slices/` for v1.1):

```bash
aws s3 ls s3://nosible-world/slices_v1_2/ --profile nosible-world
```

Inspect one date:

```bash
aws s3 ls s3://nosible-world/slices_v1_2/2015-01-01/ --profile nosible-world
```

Download one date:

```bash
aws s3 sync s3://nosible-world/slices_v1_2/2015-01-01/ \
  ./nosible-world/slices_v1_2/2015-01-01/ \
  --profile nosible-world
```

Download the full range:

```bash
aws s3 sync s3://nosible-world/slices_v1_2/ \
  ./nosible-world/slices_v1_2/ \
  --profile nosible-world
```

The full sync downloads all dates and all four files per date.

Download the sample file:

```bash
aws s3 cp s3://nosible-world/samples_v1_2.json ./ --profile nosible-world
```

## Core Lookup Contracts

The core lookup chain for semantic search is:

```text
semantic query -> embedding -> events_oai.hnsw -> event_id_u64 keys
event_id_u64 keys -> events_map.ipc row indices
row indices -> events.locs offsets -> events.ndjson full JSON records
```

For non-semantic workflows:

```text
filters/sorts/facets -> events_map.ipc row indices
row indices -> events.locs offsets -> events.ndjson full JSON records
```

Hard requirements:

1. All four files in a date directory must belong to the same slice date.
2. `events.ndjson` is newline-delimited UTF-8 JSON, one event per line.
3. `events.locs` indexes line numbers, not event IDs.
4. `events.locs` length is `number_of_events + 1`.
5. `events.locs[-1]` must equal `os.path.getsize(events.ndjson)`.
6. `events_map.ipc` has one row per event.
7. `events_map.ipc` row `i` corresponds to `events.ndjson` line `i` (event order
   is preserved). There is no `ndjson_loc` column; use the row index directly.
8. `event_id_u64` is `xxhash64(event_id, seed=0)`.
9. HNSW result keys are `event_id_u64`, not row numbers.
10. HNSW keys and `event_id_u64` must be handled as unsigned 64-bit integers.
11. Query vectors for `events_oai.hnsw` must use `text-embedding-3-large`,
    `3072` dimensions, `f32`, cosine.
12. Do not query the shipped index with a reduced dimension: `events_oai.hnsw`
    is built on the full `3072` dims, so the query must be `3072` to match. (The
    stored vectors are Matryoshka -- you may truncate them for your OWN
    similarity work; see the `oai_vector` section.)
13. L2-normalizing the query vector is neither required nor harmful. The index
    uses the cosine metric, which is magnitude-invariant, and uSearch normalizes
    internally, so a raw or an L2-normalized query returns identical results.

Important: `event_id_u64` is unsigned. Values can exceed `2^63 - 1`. Do not
store HNSW keys or `event_id_u64` in signed 64-bit columns, JavaScript numbers,
or any system that cannot preserve the full unsigned 64-bit value.

## `events.ndjson`

Format:

```text
{"event_id":"...","event":{...},"coverage":{...},...}\n
{"event_id":"...","event":{...},"coverage":{...},...}\n
...
```

Each line is one canonical NOSIBLE World event. Typical top-level keys include:

```text
event_id
version
has_tickers
event
coordinate
signals
coverage
entities
tickers
ontology
provenance
similar
oai_vector
extra
```

Common nested fields:

```text
event.title
event.description
event.date
event.language
event.country
event.region
event.continent

signals.sentiment
signals.forward_looking
signals.materiality
signals.materiality_score
signals.time_horizon

coverage.total_coverage
coverage.total_netlocs
coverage.coverage_concentration_index

entities
tickers
ontology
provenance
similar
oai_vector
extra.signals
```

`event.date` equals the slice directory name. Use it for sanity checks, not as
a join key.

### Provenance

`provenance` is a list of source records:

```json
[
  {
    "event_score": 0.9132,
    "url": "https://example.com/article",
    "title": "Source article title",
    "description": "Source summary or snippet",
    "snippets": ["best source snippet", "another event-level snippet"]
  }
]
```

Use `provenance` when you need source URLs, source titles, evidence snippets,
source descriptions, or per-source event scores. This payload is deliberately
kept in `events.ndjson`, not `events_map.ipc`.

### Tickers

`tickers` is always a list, but it may be empty: an event with no company
ticker has `tickers: []` and `has_tickers: false`. Use `has_tickers` (present on
the event and in `events_map.ipc`) to filter to ticker-bearing events.
Symbol-level and identifier-level matching require reading the full NDJSON event
and inspecting the nested `tickers` list.

For portfolio joins, prefer stable identifiers such as FIGI, ISIN, or LEI when
available. If joining by symbol, normalize case and compare against populated
`ticker_*` fields. Do not infer tickers from organization text alone.

Exchange-symbol correction: for a set of tickers whose source `ticker_eodhd`
resolved to the wrong exchange listing (for example a depositary receipt or a
secondary cross-listing rather than the intended line), the corrected symbol is
written to `ticker_serp`. In these cases `ticker_serp` is the field to trust for
the intended listing. The original `ticker_eodhd` is deliberately left
unchanged, and the security identifiers (`figi`, `isin`, `lei`) and the other
`ticker_*` fields are NOT rewritten, so for a corrected ticker they may still
describe the original (uncorrected) listing and can disagree with `ticker_serp`.
When the corrected symbol matters, read `ticker_serp`; do not assume
`ticker_eodhd`, `figi`, `isin`, or `lei` point at the same listing for those
records. This affects the subset of records that required correction; all other
tickers are unchanged.

### Ontology

`ontology` in `events.ndjson` is the full classification payload. For each
taxonomy (IAB, IPTC media topics and genre, media frame, GICS, asset class,
SDGs, and the NOSIBLE taxonomy) it carries the ranked list of candidate
classifications, best first, each with its cosine score and a `below_floor`
flag. The flat `events_map.ipc` columns expose only the single top-ranked pick
per taxonomy; read `events.ndjson` when you need the alternates, the scores, or
the below-floor detail.

### `oai_vector`

`oai_vector` is a comma-separated decimal string of `3072` float values, not a
JSON array and not base64.

Decode it only when direct vector math is needed:

```python
import numpy as np


def decode_oai_vector(value: str) -> np.ndarray:
    vector = np.asarray(value.split(","), dtype=np.float32)
    if vector.shape != (3072,):
        raise ValueError(f"expected 3072 floats, got {vector.shape}")
    return vector
```

For normal semantic search, query `events_oai.hnsw` instead of decoding every
event vector.

These embeddings are Matryoshka (MRL): the `3072` dimensions are ordered by
importance, so you can truncate each vector to a leading prefix (for example the
first `1024` or `256` values) and still get meaningful similarity, trading some
accuracy for speed and memory. If you do this, truncate consistently on both the
stored vectors and your query; for plain cosine ranking you do not need to
re-normalize, but re-normalize the truncated vectors if you use a
magnitude-sensitive metric such as dot product. This applies ONLY to your own
similarity computations or your own index. The shipped `events_oai.hnsw` is
built on the full `3072` dimensions, so any query against that index must use
the full `3072`-dim vector.

## `events.locs`

`events.locs` is not text. It is a NumPy `.npy` binary array:

```text
dtype: int64
shape: (number_of_lines + 1,)
allow_pickle: false
```

Semantics:

```python
start = locs[line_index]
end = locs[line_index + 1]
raw_line = events_ndjson_bytes[start:end]
```

Readers should trim only trailing `\n` and `\r` before JSON parsing.

## `events_map.ipc`

`events_map.ipc` is the fast path for filtering, sorting, faceting, joining
HNSW hits to event rows, and rendering lightweight list-card fields.

Core ID columns:

```text
event_id       string; canonical event id
event_id_u64   uint64; xxhash64(event_id, seed=0), same key used in HNSW
date           string; YYYY-MM-DD
```

Common display and filter columns include:

```text
title
description
language
continent
region
country
country_iso
city
lat
lng
coordinate_source
total_coverage
total_netlocs
coverage_concentration_index
has_tickers
sentiment
materiality
materiality_score
time_horizon
required_ontology_below_floor
iab_tier_1
iab_tier_2
iab_tier_3
iab_tier_4
iptc_genre
iptc_media_topic_l1
iptc_media_topic_l2
iptc_media_topic_l3
media_frame
gics_sector
gics_industry_group
gics_industry
gics_sub_industry
asset_class_main_class
asset_class_sub_class
sdgs_goal
sdgs_target
nosible_category
nosible_subcategory
nosible_event
```

The ontology columns (`iab_*`, `iptc_*`, `media_frame`, `gics_*`,
`asset_class_*`, `sdgs_*`, and `nosible_*`) hold ONLY the single top-ranked
classification per taxonomy. The full ranked list of candidate classifications
for each taxonomy, with their cosine scores and per-candidate `below_floor`
flags, is kept in `events.ndjson` under `ontology`. Read `events.ndjson` if you
need the alternates, the scores, or the below-floor detail; `events_map.ipc`
gives you the best label only.

Row order is preserved: row `i` in `events_map.ipc` is line `i` in
`events.ndjson`. There is no `ndjson_loc` column in this distribution, so use
the row index directly as the NDJSON line index.

## `events_oai.hnsw`

Production geometry:

```text
model: text-embedding-3-large
dimensions: 3072
metric: cosine / "cos"
dtype: f32
connectivity: 32
expansion_add: 256
expansion_search: 256
multi: false
```

Index keys:

```text
key = xxhash64(event_id, seed=0)
```

Search result distances are metric distances from uSearch. For cosine indexes,
smaller is better. For display, a common approximation is:

```text
similarity = 1.0 - distance
```

Do not treat the returned key as a row number. Always join:

```text
hnsw key -> events_map.ipc event_id_u64 -> row index -> events.ndjson line
```

## Minimal Python Reader

Install dependencies:

```bash
pip install numpy pyarrow xxhash "usearch>=2.19.1"
```

Open one downloaded date:

```python
import json
import os

import numpy as np
import pyarrow.ipc
import xxhash
from usearch.index import Index


day_dir = "./nosible-world/slices_v1_2/2015-01-01"
ndjson_path = f"{day_dir}/events.ndjson"
locs_path = f"{day_dir}/events.locs"
map_path = f"{day_dir}/events_map.ipc"
hnsw_path = f"{day_dir}/events_oai.hnsw"

locs = np.load(locs_path, allow_pickle=False)
assert locs.dtype == np.int64
assert int(locs[0]) == 0
assert int(locs[-1]) == os.path.getsize(ndjson_path)

with pyarrow.ipc.open_file(map_path) as reader:
    table = reader.read_all()

assert table.num_rows == len(locs) - 1
assert "event_id" in table.column_names
assert "event_id_u64" in table.column_names

event_id = table.column("event_id")[0].as_py()
event_id_u64 = int(table.column("event_id_u64")[0].as_py())
assert xxhash.xxh64_intdigest(event_id, seed=0) == event_id_u64

with open(ndjson_path, "rb") as handle:
    start = int(locs[0])
    end = int(locs[1])
    handle.seek(start)
    event = json.loads(handle.read(end - start).rstrip(b"\r\n"))

print(event["event"]["title"])

index = Index(
    ndim=3072,
    metric="cos",
    dtype="f32",
    expansion_search=256,
)
index.view(path=hnsw_path)
assert len(index) == table.num_rows
```

## End-to-End Semantic Search

Use this sequence:

1. Open `events_map.ipc`.
2. Build a map from `event_id_u64 -> row_index`.
3. Open `events.locs`.
4. Open or memory-map `events.ndjson`.
5. Open `events_oai.hnsw` with `3072`, cosine, `f32`.
6. Embed the query with OpenAI `text-embedding-3-large`.
7. Do not pass a reduced `dimensions` value; the query must be the full `3072`
   to match the shipped index.
8. Convert the embedding to contiguous `float32`.
9. Normalization is optional: the cosine index is magnitude-invariant, so an
   L2-normalized or raw query returns the same results.
10. Search HNSW for top `k`.
11. For each returned key:
    - look up `row = key_to_row[key]`
    - use `line = row` (row index equals the NDJSON line index; there is no `ndjson_loc`)
    - read `events.ndjson[locs[line]:locs[line + 1]]`
    - parse JSON

Do not scan `events.ndjson` for semantic search joins. `event_id_u64` exists to
make this join O(1) per hit.

## End-to-End Filtering

Use this sequence:

1. Open `events_map.ipc`.
2. Apply filters against flat columns.
3. Sort using flat columns.
4. Select row indices for the requested page.
5. For list-card results, render directly from `events_map.ipc`.
6. For full results, map row indices to NDJSON line indices and read
   `events.ndjson` through `events.locs`.

Examples of filters that should use `events_map.ipc`:

```text
country == "United States"
language == "en"
materiality in {"high", "medium"}
total_coverage >= 10
total_netlocs >= 3
has_tickers == true
nosible_category == "Capital Markets"
nosible_event == "Merger Proposed"
gics_sector == "Information Technology"
sentiment == "negative"
```

Examples of fields that require `events.ndjson`:

```text
provenance URLs
per-source snippets/descriptions
full entities block
full tickers block
similar events
per-taxonomy below_floor metadata
rich nested ontology payloads not flattened into events_map.ipc
```

## Validation Checklist

Run these checks before trusting a downloaded date:

1. All four required files exist.
2. `events.locs` loads with `np.load(..., allow_pickle=False)`.
3. `events.locs.dtype == np.int64`.
4. `events.locs.ndim == 1`.
5. `events.locs[0] == 0`.
6. `events.locs[-1] == os.path.getsize(events.ndjson)`.
7. `len(events.locs) - 1 == events_map.num_rows`.
8. `events_map.ipc` has `event_id` and `event_id_u64`.
9. `event_id_u64` is read as unsigned 64-bit.
10. The HNSW index opens with matching geometry.
11. A known `event_id` hashes to the corresponding `event_id_u64`.

## Common Mistakes

Every slice object lives under `s3://nosible-world/slices/<YYYY-MM-DD>/` (v1.1)
or `s3://nosible-world/slices_v1_2/<YYYY-MM-DD>/` (v1.2); the only objects at the
bucket root are this guide (`NOSIBLE_WORLD.md`) and the `samples_v1_2.json`
preview file. Do not use any other bucket
or prefix. Earlier NOSIBLE distributions used different bucket and prefix names;
those are not valid here.

Do not parse `events.locs` as text. It is a NumPy `.npy` binary file.

Do not assume an HNSW key is a row number. It is `event_id_u64`.

Do not store HNSW keys or `event_id_u64` in signed integer columns.

Do not use a different embedding model for queries. The vectors must be
compatible with `text-embedding-3-large` at `3072` dimensions.

Do not query the shipped `events_oai.hnsw` with a reduced dimension such as
`dimensions=1536`; that index is `3072`-dim and the query must match. (Truncating
the stored vectors for your own similarity work is fine -- the embeddings are
Matryoshka; see the `oai_vector` section.)

Do not treat `oai_vector` as a JSON list of floats or a base64 blob. It is a
comma-separated decimal string with `3072` values.

Do not expect `events_map.ipc` to contain provenance. It is deliberately flat.

Do not merge artifacts from different date folders. The row, key, and offset
contracts only hold within one slice.